Papers with monolingual models
Evaluating Cross-Lingual Transfer Learning Approaches in Multilingual Conversational Agent Models (2020.coling-industry)
Copied to clipboard
| Challenge: | Existing voice assistant models are developed for each region or language, requiring linear effort to develop and maintain. |
| Approach: | They propose a general multilingual model framework for natural language understanding models . they show multilingual models can reach same or better performance compared to monolingual models a . |
| Outcome: | The proposed model framework can bootstrap new language models faster and reduce effort . it can reach same or better performance compared to monolingual models across language-specific test data . |
Generalising Multilingual Concept-to-Text NLG with Language Agnostic Delexicalisation (2021.acl-long)
Copied to clipboard
| Challenge: | Concept-to-text Natural Language Generation requires a delexicalisation of the input, but this often requires that the input appears verbatim in the output text. |
| Approach: | They propose a method that uses multilingual pretrained embeddings to inflect words in their correct form during relexicalisation. |
| Outcome: | The proposed method outperforms monolingual models in concept-to-text and in low resource environments. |
Beyond the English Web: Zero-Shot Cross-Lingual and Lightweight Monolingual Classification of Registers (2021.eacl-srw)
Copied to clipboard
Liina Repo, Valtteri Skantsi, Samuel Rönnqvist, Saara Hellström, Miika Oinonen, Anna Salmela, Douglas Biber, Jesse Egbert, Sampo Pyysalo, Veronika Laippala
| Challenge: | Existing studies on register classification for web documents have limited results due to skewed datasets and low performance. |
| Approach: | They propose two new register-annotated corpora for French and Swedish . they show that deep pre-trained language models perform strongly in these languages . |
| Outcome: | The proposed models outperform existing models in English and Finnish and can match or surpass existing models. |
CMTA: COVID-19 Misinformation Multilingual Analysis on Twitter (2021.acl-srw)
Copied to clipboard
| Challenge: | myths, sensationalism, rumours and misinformation, generated intentionally or unintentionally, spread rapidly through social networks during the COVID-19 pandemic . evaluation of tweets for recognizing misinformation can create beneficial understanding to review the top quality and also the readability of online information concerning the COV-19. |
| Approach: | They propose a multilingual COVID-19 related tweet analysis method that uses a deep learning model for multilingual tweet misinformation detection and classification. |
| Outcome: | The proposed method outperforms monolingual models in the misinformation detection task and shows that it can be used to improve the quality and readability of online information. |
Mitigating Language-Dependent Ethnic Bias in BERT (2021.emnlp-main)
Copied to clipboard
| Challenge: | Ethnic bias is one of the most prevalent social stereotypes. |
| Approach: | They propose to use a multilingual model and contextual word alignment to mitigate ethnic bias in monolingual BERT for English, German, Spanish, Korean, Turkish, and Chinese. |
| Outcome: | The proposed methods alleviate ethnic bias in English, German, Spanish, Korean, Turkish, and Chinese using a multilingual model and contextual word alignment of two monolingual models. |
MEDs for PETs: Multilingual Euphemism Disambiguation for Potentially Euphemistic Terms (2024.findings-eacl)
Copied to clipboard
Patrick Lee, Alain Chirino Trujillo, Diana Cuevas Plancarte, Olumide Ojo, Xinyi Liu, Iyanuoluwa Shode, Yuan Zhao, Anna Feldman, Jing Peng
| Challenge: | Euphemisms are a linguistic device used to soften or neutralize language that may otherwise be harsh or awkward to state directly. |
| Approach: | They train a multilingual transformer model to disambiguate potentially euphemistic terms in multilingual and cross-lingual settings. |
| Outcome: | The proposed model performs better than monolingual models on the disambiguation task compared to monolingual ones in multilingual and cross-lingual settings. |
Medical Spoken Named Entity Recognition (2025.naacl-industry)
Copied to clipboard
Khai Le-Duc, David Thulke, Hung-Phong Tran, Long Vo-Dang, Khai-Nguyen Nguyen, Truong-Son Hy, Ralf Schlüter
| Challenge: | Named Entity Recognition (NER) aims to extract named entities from speech and categorise them into types like person, location, organization, etc. |
| Approach: | They present a spoken NER dataset in the medical domain using pre-trained models that are encoder-only and sequence-to-sequence. |
| Outcome: | The dataset is the largest spoken NER dataset in the world regarding the number of entity types, featuring 18 distinct types. |
Developing multilingual speech synthesis system for Ojibwe, Mi’kmaq, and Maliseet (2025.naacl-short)
Copied to clipboard
| Challenge: | In general, speech synthesis for Indigenous languages is underdeveloped compared to the majority of languages. |
| Approach: | They propose to train a multilingual model on three typologically similar languages to improve performance over monolingual models. |
| Outcome: | The proposed model can train on three similar languages with high performance and is highly competitive with self-attention architectures with higher memory efficiency. |
Cross-lingual Transfer of Monolingual Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual learning using multilingual models cast doubt on shared vocabulary and joint pre-training . et al. (2005) show that model knowledge learned in the source language enhances the learning of the target language independently of language proximity. |
| Approach: | They propose a method for transferring monolingual models to other languages through continuous pre-training and investigate their results in English. |
| Outcome: | The proposed method outperforms a model trained from scratch in the GLUE benchmark for English . it shows that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in english. |
Strong Baselines for Complex Word Identification across Multiple Languages (N19-1)
Copied to clipboard
Pierre Finnimore, Elisabeth Fritzsch, Daniel King, Alison Sneyd, Aneeq Ur Rehman, Fernando Alva-Manchego, Andreas Vlachos
| Challenge: | Complex Word Identification (CWI) is the task of identifying which words or phrases in a sentence are difficult to understand by a specific type of reader. |
| Approach: | They propose to use monolingual and cross-lingual CWI models to make predictions for languages not seen during training. |
| Outcome: | The proposed models perform as well as (or better than) most models submitted to the latest CWI Shared Task. |
An Isotropy Analysis in the Multilingual BERT Embedding Space (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have explored the advantages of multilingual pre-trained models in capturing shared linguistic knowledge. |
| Approach: | They investigate the anisotropic embedding space and outlier dimensions of the multilingual BERT model for two known issues of the monolingual models. |
| Outcome: | The proposed model has no outlier dimension and has highly anisotropic space . the results show that increasing the isotropy of multilingual space can improve its representation power and performance, similar to what had been observed for monolingual CWRs on semantic similarity tasks. |
A Conditional Generative Matching Model for Multi-lingual Reply Suggestion (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for multilingual RS are limited by capacity and data distribution skew . we propose Conditional Generative Matching models (CGM) to overcome these challenges . |
| Approach: | They propose Conditional Generative Matching models to address multilingual RS challenges . they use expressive message conditional priors, mixture densities and latent alignment . results exceed ROUGE scores by 10% on average, and 16% for low resource languages . |
| Outcome: | The proposed model exceeds baselines in relevance by 10% on average and 16% for low resource languages. |
DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects (2026.acl-long)
Copied to clipboard
| Challenge: | Current disinformation detection systems are predominantly developed and evaluated on Standard American English (SAE) . however, their robustness to dialectal variation is unexplored. |
| Approach: | They propose a benchmark for evaluating disinformation detection robustness across 50 English dialects . they use multi-value's linguistically-grounded transformations to introduce D-CUBE (Dialectal Disinformation Detection Corpus) |
| Outcome: | The proposed model outperforms zero-shot LLMs in human-written dialects while AI-generated content remains stable. |
Polyglots or Multitudes? Multilingual LLM Answers to Value-laden Multiple-Choice Questions (2026.eacl-long)
Copied to clipboard
| Challenge: | Multiple-choice questions (MCQs) are used to assess knowledge, reasoning abilities, and even values encoded in large language models. |
| Approach: | They propose to test whether multilingual LLMs are consistent in their responses across languages . they also use human-translated questions aligned in 8 European languages to test their robustness . |
| Outcome: | The proposed corpus of questions is aligned in 8 European languages and compared with previous studies. |
FinGPT: Large Generative Models for a Small Language (2023.emnlp-main)
Copied to clipboard
Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, Sampo Pyysalo
| Challenge: | Neural language models excel in many tasks in NLP but are limited to smaller languages. |
| Approach: | They propose two approaches to pretrain large language models for Finnish . they train seven monolingual models from scratch and use Finnish as pretraining data . |
| Outcome: | The proposed model is based on a dataset of Finnish web crawls, news, social media and eBooks. |
The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild (2022.lrec-1)
Copied to clipboard
| Challenge: | GINCO is a new training dataset for automatic genre identification based on 1,125 crawled Slovenian web documents that consist of 650,000 words. |
| Approach: | They propose to use 1,125 crawled Slovenian web documents to train a new genre classification system based on a GINCO training dataset . |
| Outcome: | The proposed classifiers perform better on the 1,125 crawled Slovenian web documents than the existing models and achieve higher scores on the task. |
Attention on Multiword Expressions: A Multilingual Study of BERT-based Models with Regard to Idiomaticity and Microsyntax (2025.findings-naacl)
Copied to clipboard
| Challenge: | Specifically, models fine-tuned on semantic tasks tend to distribute attention to idiomatic expressions more evenly across layers. |
| Approach: | They analyze attention patterns of encoder-only models towards two distinct types of Multiword Expressions (MWEs) idioms present challenges in semantic non-compositionality, while MSUs demonstrate unconventional syntactic behavior that does not conform to standard grammatical categorizations. |
| Outcome: | The proposed models show that fine-tuned models allocate attention to idiomatic expressions more evenly across layers. |
Contrastive Aligned Joint Learning for Multilingual Summarization (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing summarization systems for multilingual text summarizing are limited due to the lack of large-scale data in multiple languages. |
| Approach: | They propose a multilingual summarization system that can understand documents in multiple languages and generate summaries in the corresponding language. |
| Outcome: | The proposed model improves over monolingual models in all languages and transferable to other languages. |
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Using pretraining data, we find that a designated monolingual tokenizer plays an equally important role in the downstream performance of the model. |
| Approach: | They propose to compare pretrained multilingual models with their monolingual counterparts on a set of five diverse monolingual downstream tasks. |
| Outcome: | The proposed models offer previously unmatched performance in all NLP tasks. |
Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models require a more expressive vocabulary to represent all languages . however, increasing the vocabulary size significantly slows down the pre-training speed . |
| Approach: | They propose an algorithm VoCap to determine the desired vocabulary capacity of each language. |
| Outcome: | The proposed algorithm improves cross-lingual model pre-training while reducing side effects of increasing vocabulary size. |
Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models that represent different senses of words in context are not accurate for polysemous words. |
| Approach: | They propose a multilingual dataset that evaluates the ability of models to accurately represent different lexical-semantic relations such as homonymy and synonymy. |
| Outcome: | The proposed models can disambiguate homonyms in context, but fail to represent words with different senses when occurring in similar sentences. |
Structure-Level Knowledge Distillation For Multilingual Sequence Labeling (2020.acl-main)
Copied to clipboard
| Challenge: | Existing multilingual models still underperform individual monolingual models due to model capacity limitations. |
| Approach: | They propose to distill the structural knowledge of several monolingual models (teachers) to the unified multilingual model (student). |
| Outcome: | The proposed model outperforms strong baseline models and teacher models on 4 multilingual tasks with 25 datasets and has stronger zero-shot generalizability. |
Translation Deserves Better: Analyzing Translation Artifacts in Cross-lingual Visual Question Answering (2024.findings-acl)
Copied to clipboard
ChaeHun Park, Koanho Lee, Hyesu Lim, Jaeseok Kim, Junmo Park, Yu-Jung Heo, Du-Seong Chang, Jaegul Choo
| Challenge: | Recent studies have employed machine translation systems for cross-lingual VQA tasks . however, translated texts contain unique characteristics distinct from human-written ones, referred to as translation artifacts. |
| Approach: | They propose a machine translation system that can train models in multiple languages . they propose augmentation strategies that reduce translation artifacts in translated texts . |
| Outcome: | The proposed approach reduces translation artifacts in models across languages and languages. |
MonoByte: A Pool of Monolingual Byte-level Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies have shown that multilingual models can achieve zero-shot cross-lingual performance on various NLP tasks, but due to the cost of pretraining, they often use public models with limited budgets. |
| Approach: | They propose to use tokenized models to test cross-lingual ability in multilingual and monolingual corpora. |
| Outcome: | The results show that models pretrained on multilingual and even monolingual corpora perform better than models pre-trained on SOTA models. |
Are Pretrained Multilingual Models Equally Fair across Languages? (2022.coling-1)
Copied to clipboard
| Challenge: | Pretrained multilingual language models can help bridge the digital language divide, enabling high-quality NLP models for lower-resourced languages. |
| Approach: | They propose to use a multilingual dataset to examine whether multilingual models are equally fair across languages. |
| Outcome: | The proposed model enables apples-to-apples comparison across languages of group disparities in multilingual language models. |
Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Multilingual transformers have been shown to have remarkable transfer skills in zero-shot settings. |
| Approach: | They investigate cross-lingual transfer abilities of XLM-R for Chinese and English natural language inference using a large scale Chinese dataset. |
| Outcome: | The proposed model trains on Chinese and English natural language inference datasets. |
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing sentence embeddings models are monolingual, and only for English . a new method allows to create multilingual versions from monolingual models . |
| Approach: | They propose a method to extend existing sentence embedding models to new languages . they use a translated sentence to generate sentence embeds for the source language . |
| Outcome: | The proposed method improves accuracy for multilingual setups and languages. |
Measuring Cross-lingual Transfer in Bytes (2024.naacl-long)
Copied to clipboard
| Challenge: | Multilingual pretraining models can transfer knowledge to target languages with minimal or no examples . underlying mechanisms for this transfer remain unclear, with hypotheses ranging from language contamination to syntactic similarity. |
| Approach: | They conducted an experiment to investigate whether multilingual models transfer knowledge to target languages . they found that models initialized from diverse languages perform similarly to a target language . |
| Outcome: | a new study shows that models initialized from diverse languages perform similarly to a target language in a cross-lingual setting. |
Multilingual Models for ASR in Chibchan Languages (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing algorithms for low resource-intensive languages are not available for these languages . a paper comparing the performance of different models and algorithms for these extremely low resource languages is presented. |
| Approach: | They propose to fine-tune four ASR algorithms to create monolingual models for Bribri and Cabécar . they then use the best performing algorithm to train joint and transfer learning models for both languages . |
| Outcome: | The proposed algorithms are effective in both Bribri and Cabécar, but especially in Bribri. |
Evaluating Self-Supervised Speech Representations for Indigenous American Languages (2024.lrec-main)
Copied to clipboard
| Challenge: | a recent study focused on the use of self-supervised learning to learn speech representations for indigenous languages . aaron e. scott: the vast linguistic diversity represented by indigenous languages remains unexplored . by expanding the scope of language processing to include indigenous languages, we can foster linguistic inclusivity, he says . |
| Approach: | They benchmark the efficacy of large-scale self-supervised learning models on indigenous American languages. |
| Outcome: | The proposed model can generalize to real-world data, showing strong performance . evaluators found that the model performed better than monolingual models on indigenous languages . |
Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models memorize large amounts of information, raising significant safety concerns. |
| Approach: | They propose an approach to machine unlearning for multilingual language models that selectively erases information across different languages while maintaining overall performance. |
| Outcome: | The proposed approach is compared with existing unlearning baselines and set a new standard for secure and adaptable multilingual language models. |
From English to Code-Switching: Transfer Learning with Strong Morphological Clues (2020.acl-main)
Copied to clipboard
| Challenge: | Linguistic code-switching (CS) is an understudied area in natural language processing . lack of resources and annotated data makes it difficult to strive for progress in CS-related tasks. |
| Approach: | They propose a method to adapt monolingual models to code-switched text in various tasks . they transfer English knowledge from a pre-trained ELMo model to different code-paired languages . |
| Outcome: | The proposed method outperforms multilingual BERT and homologous CS-unaware models and provides state-of-the-art in CS tasks. |
Pre-training and Evaluating Transformer-based Language Models for Icelandic (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained models obtain state-of-the-art performance on a wide variety of NLP tasks, including Question Answering (QA), Named Entity Recognition (NER), Part-of Speech (POS) tagging and Automatic Text Summarization (ATS). |
| Approach: | They pre-train four types of monolingual ELECTRA and ConvBERT models and compare them to a previously trained monolingual RoBERTa model and multilingual mBERT model. |
| Outcome: | The models outperform a multilingual model on four downstream tasks. |
Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages (2023.findings-emnlp)
Copied to clipboard
Elena Ruzzetti, Federico Ranaldi, Felicia Logozzo, Michele Mastromattei, Leonardo Ranaldi, Fabio Zanzotto
| Challenge: | a recent study examined how models for typologically similar languages encode structural information. |
| Approach: | They propose to layer-wise compare transformers for typologically similar languages to observe similarities . they use a domain adaptation on semantically equivalent texts to measure similarity . |
| Outcome: | The proposed model outperforms all other models on unseen sentences . the proposed model is based on a typologically similar language . |
Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using multilingual models, we find that treating languages in isolation obscures the true patterns of memorization. |
| Approach: | They propose a graph-based correlation metric that incorporates language similarity to analyze cross-lingual memorization. |
| Outcome: | The proposed model incorporates language similarity to analyze cross-lingual memorization in 95 languages. |
Efficient Multilingual Language Model Compression through Vocabulary Trimming (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual language models (LMs) have become a powerful tool in NLP, especially for non-English languages. |
| Approach: | They propose a method to reduce a multilingual LM vocabulary to a target language by deleting potentially irrelevant tokens from its vocabulary. |
| Outcome: | The proposed method can retain the original performance of the multilingual LM while being considerably smaller in size than the original model. |
Multilingual Turn-taking Prediction Using Voice Activity Projection (2024.lrec-main)
Copied to clipboard
| Challenge: | a monolingual model does not make good predictions when applied to other languages, but a multilingual model is able to discern the language of the input signal. |
| Approach: | They propose to use a multilingual voice activity projection model to predict voice activities of spoken dialogue participants in English, Mandarin, and Japanese data. |
| Outcome: | The proposed model predicts the upcoming voice activities of participants in dyadic dialogue on multilingual data, encompassing English, Mandarin, and Japanese. |